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Abstract 
In this 3-year prospective study, we tested the extent to which school-, practice-, and district- 
level variables predicted sustained implementation for schools in various stages of 
implementation of school-wide positive behavioral interventions and supports (SWPBIS) Tier 1 
(universal) systems. Staff from 860 schools in 14 U.S. states completed a research-validated 
measure of factors associated with sustained implementation of school interventions during Y ear 
1 of this study. Analyses included multi-group structural equation modeling of school and district 
implementation fidelity data. Results indicated that adequate implementation fidelity and better 
Team Use of Data for decision making in Study Year | were the strongest predictors of sustained 
implementation in Year 3. In addition, the number of other schools in the district adopting 
SWPBIS was a similarly strong predictor. A critical mass of schools implementing was also 
predictive, especially for schools earlier in implementation. School characteristics were not 


predictive, except for grade levels served, which was an inconsistent predictor by stage. 
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Factors Predicting Sustained Implementation of a Universal Behavior 
Support Framework 

For over 40 years, researchers have stressed the importance of sustaining evidence-based 
practices (EBPs), those with demonstrated evidence of effectiveness, for improving outcomes for 
students (Berman & McLaughlin, 1976). Yet until recently, few large-scale studies had 
examined factors influencing the implementation and sustainability of EBPs in schools (Fixsen, 
Blase, & Fixsen, 2017). One prominent EBP is school-wide positive behavioral interventions and 
supports (SWPBIS; Horner et al., 2014), an evidence-based, multi-tiered framework for 
implementing school interventions to improve school climate and social and emotional 
competence that is currently being implemented in over 25,000 schools in the U.S. (Center on 
Positive Behavioral Interventions and Supports, 2018). The multi-tiered component of SWPBIS 
maps onto a public health model. Tier 1, or universal supports, and interventions and strategies 
delivered to all students in the school. These practices are preventative in nature and easy to 
implement, such as defining, teaching, and rewarding use of a set of school-wide behavior 
expectations and using clear guidelines for addressing behavior concerns instructionally. Tier 2, 
or targeted supports, include interventions delivered to a smaller number of students who need 
strategic supports in addition to those delivered at Tier 1. In contrast to the standardized 
interventions at Tiers | and 2, Tier 3, or intensive supports, are individualized to student need. 
Research has shown a range of positive effects of implementing SWPBIS, including increased 
student emotional regulation, decreased use of exclusionary discipline, decreased racial/ethnic 
disproportionality in exclusionary discipline, and substantial fiscal benefits compared to costs 
(Bradshaw, Waasdorp, & Leaf, 2012; McIntosh, Gion, & Bastable, 2018; Swain-Bradway, 


Lindstrom Johnson, Bradshaw, & McIntosh, 2017). 
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Although it is widely accepted that implementing SWPBIS or any other practice with 
adequate fidelity often requires a certain amount of resources, research has documented that 
sustaining EBPs, regardless of effectiveness, is the exception and not the rule (Coburn, Russell, 
Kaufman, & Stein, 2012; Sindelar, Shearer, Yendol-Hoppey, & Liebert, 2006). Therefore, there 
remains a need to examine how practices can be sustained by addressing both malleable (e.g., 
provision of coaching) and nonmalleable barriers (e.g., community poverty; Hume & McIntosh, 
2013). Several studies have identified practice, school, and district-level variables that increase 
the likelihood that school-level practices will sustain over time. 

Variables Related to Sustained Implementation 

School-level variables. Influences at the school level, most specifically school 
demographic characteristics, have been noted as important to the sustainability of EBPs 
(Bradshaw & Pas, 2011; Forman et al., 2013). However, research findings in this area are mixed. 
Some studies have found school characteristics, such as serving a large population of students 
that qualify for free or reduced lunch (1.e., community poverty), to be associated with greater risk 
of schools abandoning EBPs (Bradshaw & Pas, 2011; Gottfredson, Jones, & Gore, 2002). Others 
found that the abandonment risks associated with community poverty were no longer significant 
when practice-level factors, such as providing standardized materials for ease of implementation, 
were in place (Payne & Eckert, 2010). More consistency is found amongst studies that examined 
the specific nonmalleable characteristics of grade levels served and school locale (e.g., city, 
suburb, town, and rural) in schools implementing SWPBIS. These studies have identified 
adequate implementation is reached more quickly in elementary schools than middle and high 


schools (potentially due to the size and complexity of high schools) and in suburban 
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neighborhoods than cities (potentially due to higher rates of turnover; Bradshaw & Pas, 2011; 
McIntosh et al., 2013; Nese et al., 2016; Schaper, McIntosh, & Hoselton, 2016). 

Practice-level variables. Practice-level variables are specific aspects of the practice itself 
or activities related its implementation. One practice-level variable identified in the literature is 
the use of data for decision making by intervention teams (Fixsen, Blase, Duda, Naoom, & Van 
Dyke, 2010; Johnson, Collins, Shamblen, Kenworthy, & Wandersman, 2017). Using a measure 
to assess factors critical to the sustainability of behavior practices in schools, the School-wide 
Universal Behavior Sustainability Index: School Teams (SUBSIST; McIntosh, Doolittle, 
Vincent, Horner, & Ervin, 2009), McIntosh and colleagues (2013) found that teams who were 
able to collect, disaggregate, and share data with their staff were more likely to implement 
SWPBIS with adequate fidelity, or as intended by developers. These findings illustrate the 
importance of teams and how their use of data can enhance implementation of SWPBIS 
(McIntosh et al., 2013). Another practice-level variable found to be important for sustained 
implementation is the speed at which practices can be installed with fidelity. Schaper, McIntosh, 
and Hoselton (2016) found that reaching adequate levels of fidelity quickly was important for 
sustained implementation. These findings also align with community health research by Saldana 
and colleagues (2012), who found that the proportion of implementation activities completed and 
the duration of time completing them predicted future implementation. 

District-level variables. District-level variables, either contextual variables (e.g., 
community need) or professional development (i.e., training and coaching) provided to help 
schools implement EBPs, have been documented as important to sustainability. In a study of 
5331 schools implementing Tier 1 SWPBIS, McIntosh, Mercer, Nese, and Ghemraoui (2016) 


found that elementary schools, larger schools, schools in districts with more schools already 
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implementing, and those starting within a larger initial district cohort were more likely to be in 
the sustaining groups. Similarly, two randomized controlled effectiveness trials of school-wide 
practices found faster implementation among schools with strong organizational health and 
district support (Bradshaw, Koth, Thornton, & Leaf, 2009; Horner et al., 2009). It is important to 
note that district supports are likely to vary depending on available resources and the extent to 
which the practice is used by nearby professionals (examined in community health by Klest, 
2014). District teams may be more likely to invest in professional development for an initiative if 
it is perceived to be effective, implemented longer, and embedded within existing leadership, 
organizational, and training and coaching systems to support its implementation (Fixsen, Naoom, 
Blase, Friedman, & Wallace, 2005). 
Differential Importance by Stage of Implementation 

Adelman and Taylor (1997) identified a series of stages of implementation (1.e., Creating 
Readiness, Initial Implementation, Institutionalization, and Ongoing Evolution) that describe 
how teams can implement new initiatives in ways that are durable and scalable. In the Creating 
Readiness stage, school staff work to build interest and support for the incoming initiative before 
implementation. Initiatives then move into the Initial Implementation stage as the school 
leadership team prepares for and begins to install the new initiative. The Institutionalization stage 
occurs once school teams have fully implemented the initiative for an extended period, 
developed long-term ownership, and overcome barriers to their continued use of the initiative. 
Ongoing Evolution occurs when initiatives experience contextual shifts and renewal from school 
teams based on continual evaluation and improvement in quality and support (Adelman & 
Taylor, 1997). Because initiatives are carried through these stages of implementation (sometime 


recursively), schools may have different needs depending on the stage of implementation 
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(Saldana et al., 2012). For example, Fixsen and colleagues (2005) hypothesized that school 
implementers in the Ongoing Evolution stage need support to buffer the initiative against the 
effects of turnover, which may increase after an extended period of time beyond Initial 
Installation and school team training. This type of support is likely to vary considerably from the 
support needed in Initial Implementation (Fixsen et al., 2005). 

The Present Study 

Much of the research examining the sustainability of EBPs has been short term, 
retrospective, or focused on variables associated with sustained implementation (1.e., concurrent 
association), but not malleable and nonmalleable factors that predict sustained implementation 
(Fixsen et al., 2010; Han & Weiss, 2005). Furthermore, much of the research has examined 
sustained implementation shortly after initial implementation, but not at various stages after 
implementation (e.g., 3 to 5 years after initial implementation). 

The following study addresses these gaps in the research by examining 3 years of 
prospective longitudinal data from 860 schools in the U.S. to answer the following research 
questions: 

1. What school-level, practice-level, and district-level variables predict sustained 
implementation of SWPBIS Tier | systems after 3 years? 

2. To what extent does stage of implementation (i.e., Initial Implementation, 
Institutionalization, and Ongoing Evolution) affect the prediction of sustained 
implementation? 

Method 


Participants and Settings 
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The sample included 860 schools across 14 states implementing SWPBIS as part of their 
respective state or district initiatives, with natural variations in levels of training and coaching as 
could be expected in large-scale implementation. Of the 686 schools that provided data on 
training in the first year of the study, 93% received formal SWPBIS training, and of the 638 
schools that reported on coaching received, 79% reported receiving at least some coaching. One 
individual familiar with each school’s SWPBIS systems (e.g., School SWPBIS team member, 
district coach) participated. Participant roles were the following: SWPBIS team leaders (61%), 
school administrators (24%), other faculty or staff (8%), or district coaches (5%). The self- 
reported number of years implementing SWPBIS was used as a proxy to categorize schools into 
three stages of implementation: Initial Implementation (0 to 1 years), Institutionalization (2 to 4 
years), and Ongoing Evolution (5 or more years) as defined by Adelman and Taylor (1997). 
Table 1 presents National Center for Education Statistics (NCES) data (available for 98%, n = 
847, of the sample) for schools in the first year of the study (not necessarily their first year of 
implementation). 

Measures 

School-level variables. School characteristics were used to assess each schools’ relation 
with sustained implementation. These data, obtained from the NCES database, included grade 
levels served (coded dichotomously as elementary or secondary), school enrollment, urbanicity 
(on a scale of 1 to 4, using the federal categories of rural, town, suburb, and city), proportion of 
non-white students, and proportion of students receiving free and/or reduced price lunch. 

Practice-level variables. Practice-level variables included measures of fidelity of 
implementation of SWPBIS and a validated measure of factors predicting sustained 


implementation of universal behavior support interventions. These measures are described 
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briefly below, with more psychometric details regarding the linking of fidelity criterion scores 
available from Mercer, McIntosh, and Hoselton (2017). 

Fidelity of Implementation. One of five research-validated Tier 1 SWPBIS fidelity of 
implementation measures was used to quantify percent implementation in Study Years | (as a 
predictor variable) and 3 (as an outcome variable). 

School-wide Evaluation Tool. For schools using the School-wide Evaluation Tool (SET; 
Sugai, Lewis-Palmer, Todd, & Horner, 2001), the SET fidelity criterion for adequate 
implementation of SWPBIS at Tier 1 (80% on all school-wide expectations taught and 80% 
overall implementation) was used. The SET is a 28-item measure with strong psychometric 
properties, including internal consistency of .96, mean interrater reliability between 98.4% and 
100%, and mean test-retest reliability between 89.9% and 100% (Horner, Todd, Lewis-Palmer, 
Irvin, & Sugai, 2004). 

Tiered Fidelity Inventory. If schools did not complete the SET, scores from the Tier 1 
scale of the Tiered Fidelity Inventory (TFI; Algozzine et al., 2014) were used. The TFI is used as 
a self-assessment measure for each of the SWPBIS tiers (i.e., Tier 1, Tier 2, and Tier 3). The Tier 
1 scale is comprised of three subscales measuring school teams, implementation, and evaluation 
of SWPBIS. The TFI Tier 1 fidelity criterion is 70%. The TFI has strong psychometric properties 
for content validity (means across tiers rated by an expert panel of reviewers = 91% to 93% for 
reliability agreement, 96% for item validity, 95% for factor structure, and 89% for item scoring), 
internal consistency (a = .87 to .98 across tiers and .96 overall), test-retest reliability (7 = .99), 
and interrater reliability across raters, tiers, and items (7 = .99), and evidence of a strong factor 


structure (Massar, McIntosh, Mercer, & Hoselton, in press; McIntosh et al., 2017). 
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Benchmarks of Quality. If schools did not complete the SET or the TFI, the School-wide 
Benchmarks of Quality (BoQ; Kincaid, Childs, & George, 2005) was used, with a 70% fidelity 
criterion. The BoQ is a 53-item measure comprised of 10 subscales of elements important for 
Tier | SWPBIS implementation (e.g., data entry and analysis established, faculty commitment). 
It has strong psychometric properties, including internal consistency of .96, test-retest of .94, 
interrater reliability .87, and concurrent validity of .63, when total scores on the BoQ were 
correlated with total scores on the SET (Cohen, Kincaid, & Childs, 2007; Mercer et al., 2017). 

PBIS Self Assessment Survey. If schools did not complete the SET, TFI, or BoQ, the 
PBIS Self-Assessment Survey (SAS; Sugai, Horner, & Todd, 2000) was used, with an 80% 
fidelity criterion. The 43-item survey is used by school staff to guide the implementation of 
SWPBIS at Tier 1 across multiple school systems (i.e., school-wide, non-classroom, classroom, 
and individual student intensive support). Internal consistency reported across two studies 
(Hagan-Burke et al., 2005; Safran, 2006) was high (a range = .85 to .94), with strong construct 
validity (7 = .75) when correlated with the SET (Horner et al., 2004). 

Team Implementation Checklist. For schools who did not complete the SET, TFI, BoQ, 
or SAS, fidelity scores from the Team Implementation Checklist (TIC; Sugai, Horner, & Lewis- 
Palmer, 2001) were used, with an 80% fidelity criterion. The TIC is used by teams for progress 
monitoring and action planning during implementation of specific features (e.g., establish 
commitment, establish and maintain a team) to assess overall SWPBIS implementation at Tier 1. 
Teams are encouraged to complete the TIC quarterly to re-evaluate progress for reaching 
adequate implementation. Both versions of the TIC have strong internal consistency, with ordinal 
alphas of .93 to .94 for overall implementation (McIntosh, Mercer, Nese, Strickland-Cohen, & 


Hoselton, 2016). 
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Coding of fidelity. Schools were categorized dichotomously as either at or below 
adequate fidelity of implementation based on published criteria using one of the measures. If 
schools completed more than one, the measure with the strongest psychometric properties was 
used in a cascading logic, in the order presented above (McIntosh et al., 2013). Additional 
support for this decision comes from a linking study showing moderate to strong concurrent 
validity among these measures (Mercer et al., 2017). At Year 1, most schools completed a SET 
(59%), followed by the BoQ (27%), SAS (13%), and TIC (2%). At Year 3, most schools 
completed a BoQ (49%), followed by the SET (32%), TFI (10%), SAS (8%), and TIC (1%). 

Practice sustainability. The SUBSIST is a research instrument used to assess features 
predicting sustained implementation of SWPBIS. School personnel rate the extent to which 
school and district-level variables (39 total) are in place in their school using a 4-point Likert- 
type scale (from | = not true to 4 = true). The SUBSIST displays evidence of strong content 
validity, with an expert panel content validity index of .95, internal consistency of .87, interrater 
reliability of .95, and two-week test-retest reliability of .96 (McIntosh et al., 2011). Further 
validation through several factor analytic studies demonstrated strong prediction of sustained 
SWPBIS implementation (7 = .45 to .68), and indicated a four-factor structure, including two 
school-level factors (School Priority, a = .94 and Team Use of Data, a = .94) and two district- 
level factors (District Priority, a = .71 and Capacity Building, o = .74) across multiple samples 
(McIntosh et al., 2011; McIntosh et al., 2013; Mercer, McIntosh, Strickland-Cohen, & Horner, 
2014; Runge, Gorlaski, & Wagner, 2014). As an example, the Team Use of Data factor contains 
11 items and assesses items related to the frequency (e.g., “Data are presented to all school 
personnel at least four times per year”) and extent to which school teams use data to improve 


implementation (e.g., “Data are used for problem solving, decision making, and action planning 
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[to make SWPBIS more effective and/or efficient]’’). The factor structure has been shown to be 
invariant across stage of implementation (Mercer et al., 2014). 

District-level variables. Two variables were calculated to represent the implementation 
context of the school district: Critical Mass and Initiative Health. 

Critical Mass. Critical Mass was operationalized as the proportion of schools in the 
district implementing SWPBIS in the year before Study Year 1 (i.e., Year 0). Schools reporting 
data to the OSEP Technical Assistance Center on Positive Behavioral Interventions and Supports 
(PBIS) on one or more fidelity assessments, regardless of the specific fidelity scores obtained, 
were considered to be implementing SWPBIS. 

Initiative Health. {nitiative Health is defined as the extent to which the initiative was 
increasing or decreasing in use across the district. For analyses, Initiative Health was 
operationalized as the change in the proportion of schools in the district newly adopting or 
abandoning SWPBIS in Year 1. Specifically, the number of schools implementing SWPBIS in 
Year 0 was subtracted from the number implementing in Year 1, with the difference divided by 
the number of schools in the district. Positive scores indicate that more schools were adopting 
SWPBIS (i.e., expansion of the initiative), whereas negative scores indicate that schools were 
abandoning SWPBIS (i.e., contraction of the initiative). 

Procedure 

Data for the current study came from a 3-year, federally-funded project examining 
implementation and sustainability of SWPBIS. For recruitment, state SWPBIS teams shared 
details at training events or sent emails to schools either implementing or preparing to implement 
SWPBIS through state or district training events. One member of each school team participated 


by completing the SUBSIST online. Extant SWPBIS fidelity of implementation data were 
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obtained from school teams’ submission of scores to the Center on PBIS. School characteristic 
data were obtained through the NCES website. 
Data Analyses 

We used multi-group structural equation modeling in Mplus 7.4 (Muthén & Muthén, 
1998-2017) to determine the extent to which school-, practice-, and district-level variables 
predicted adequate implementation in Year 3, using the robust weighted least squares (WLSMV) 
estimator to address the ordered, categorical response format of the SUBSIST. We used the 
Mplus COMPLEX command (Asparouhov, 2005) to adjust standard errors for non-independence 
due to nesting of schools in districts. We evaluated model fit with the Comparative Fit Index 
(CFI), Tucker-Lewis Index (TLI), and Root Mean Square Error of Approximation (RMSEA) 
considering recommendations that CFI ~> .95, TLI ~> .95, and RMSEA ~< .06 indicate 
acceptable fit (Hu & Bentler, 1999). We used multiple imputation (1,000 datasets) to avoid 
additional, restrictive assumptions of WLS estimation in the presence of missing data 
(Asparouhov & Muthén, 2010). 

To evaluate the research questions, we regressed the binary Year 3 fidelity variable on 
school-level predictors, practice-level predictors, and district-level predictors. We entered most 
predictors as observed variables; we specified School Priority, Team Use of Data, District 
Priority, and Capacity Building as latent variables with categorical indicators, given that 
respondents completed multiple items per SUBSIST factor. A prior study with Year 1 data from 
the same sample (Mercer et al., 2014) reported evidence of strong measurement invariance (i.e., 
equal loadings and thresholds) for the Team Use of Data, District Priority, and Capacity Building 


factors across SWPBIS implementation stages. For School Priority, the factor loadings and 


PREDICTING SUSTAINED IMPLEMENTATION 14 


thresholds differed across groups for one item; thus, we allowed them to vary for that item in the 
current analyses with all others constrained to equality across groups. 

The planned multi-group model with all practice-level variables entered as predictors 
exhibited strong multicollinearity among two SUBSIST factors (7 = .80 to .86 across groups). To 
reduce multicollinearity, we entered the SUBSIST factor with the strongest zero-order 
correlation with Year 3 fidelity as a predictor, allowing the remaining factors to covary with 
Year 3 fidelity. To facilitate interpretation, we present odds ratios (ORs), comparing values of 1 
vs. 0 for binary predictors or a 1 SD change for continuous predictors. In addition, we present R? 
values, representing proportions of variance in the continuous latent response variable underlying 
the binary Year 3 fidelity criterion variable (McKelvey & Zavoina, 1975). 

Results 

Means, by implementation group, for the criterion and predictor variables are presented 
in Table 2. Although similar percentages of schools met or exceeded the criterion for adequate 
implementation in Year 3 of the study across implementation groups (60 to 64%), greater 
percentages of schools were adequately implementing in Year 1 for groups implementing longer. 
In the Initial Implementation group, 46% of schools were adequately implementing in Year 1, 
compared to 65% in the Institutionalization and 78% in the Ongoing Evolution groups. 

Overall fit for the multi-group predictive structural equation model was good based on 
the RMSEA and near the recommended targets for acceptable fit on the CFI and TLI (Hu & 
Bentler, 1999), y” (3231) = 4048.61, p < .001; CFI = .943, TLI = .940, RMSEA = .030. To 
maximize generalizability, no post-hoc model modifications were explored. Detailed results for 
the multi-group predictive structural equation model are presented in Table 3 and represented in 


Figure 1. Only one school-level variable, school level (i.e., being an elementary school) was a 
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statistically significant predictor of Year 3 fidelity and in only the Institutionalization stage (p < 
.001, OR = 2.22). Two practice-level variables were predictors of Year 3 fidelity across 
implementation stages: fidelity in Year | (Initial Implementation: p < .001, OR = 3.64; 
Institutionalization: p < .001, OR = 3.77; Ongoing Evolution: p = .004, OR = 4.41) and greater 
SWPBIS Team Use of Data (Initial Implementation: p = .004, OR = 1.73; Institutionalization: p 
= .018, OR = 1.36; Ongoing Evolution: p < .001, OR = 1.82). No other practice-level variables 
had statistically significant associations with Year 3 fidelity after accounting for the other 
predictors. Both district-level variables were statistically significant predictors of Year 3 fidelity, 
but with some differences across SWPBIS implementation stages. Critical Mass predicted Year 3 
fidelity in Initial Implementation (p = .006, OR = 2.25) and Institutionalization (p < .001, OR = 
2.42), but not in Ongoing Evolution (p = .158). By contrast, Initiative Health predicted Year 3 
fidelity across all implementation stages (Initial Implementation: p < .001, OR = 4.18; 
Institutionalization: p < .001, OR = 2.17; Ongoing Evolution: p = .027, OR = 1.71). Collectively, 
the predictors explained similar proportions of variance in Year 3 fidelity (Initial 
Implementation: R’ = .45; Institutionalization: R? = .46; Ongoing Evolution: R? = .42). 
Discussion 

The purpose of this prospective study was to identify the strongest school, practice, and 
district predictors of sustained implementation of a school-wide universal behavior framework 
after 3 years. We also sought to test whether predictors varied by years implementing, as a proxy 
for stage of implementation. Results showed that the practice-level variables of fidelity of 
implementation in Study Year 1 and Team Use of Data were the strongest predictors of sustained 
implementation for schools in all three stages of implementation; these findings are notable 


given the prospective design and large, multi-state sample of schools in this study. 
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The findings were encouraging. Across all groups, nearly two-thirds of schools were still 
implementing SWPBIS after three years, which was unique for educational initiatives, especially 
because implementation occurred in typical school systems, outside of university-supported 
trials. Moreover, the predictors in the model accounted for a significant and substantial degree of 
variance in sustained outcomes, indicating that they represent important targets for enhancing 
sustainability. 

One key finding was that non-malleable school characteristics (e.g., poverty, enrollment) 
were not strong predictors of sustained implementation. Building on previous SWPBIS research 
using cross-sectional and retrospective designs (McIntosh et al., 2013; McIntosh, Mercer, Nese, 
Strickland-Cohen, et al., 2016), grade level was the only school characteristic found to be a 
significant predictor (elementary schools were more likely to sustain), and only at the 
Institutionalization stage of implementation (i.e., schools implementing from 2 to 4 years). 
Notably, grade level was not predictive for schools that had been implementing for 5 or more 
years, indicating that once secondary schools have been implementing for some time, they were 
as likely to sustain as elementary schools (Bohanon et al., 2006). 

In contrast, practice-level variables—implementing with fidelity in Year 1 and SWPBIS 
Team Use of Data—were stronger predictors of sustained implementation at Year 3. The finding 
regarding fidelity is unsurprising, given that it is the same variable as the outcome, but measured 
at the outset of the study. At the same time, it confirms retrospective research showing schools 
that reaching the fidelity criterion early in implementation are more likely to sustain (McIntosh, 
Mercer, Nese, Strickland-Cohen, et al., 2016). It highlights a key mechanism of sustainability: 


adequate implementation of critical features of SWPBIS improve student outcomes, which 
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reinforces the implementation behaviors of school personnel (Andreou, McIntosh, Ross, & 
Kahn, 2015; McIntosh, Kelm, & Canizal Delabra, 2016). 

The next strongest predictors were at the district level. Initiative Health was a 
consistently significant predictor of sustained implementation. Consistent with community health 
research (Klest, 2014), Critical Mass (i.e., the proportion of schools implementing SWPBIS 
before Year | of the study) predicted sustained implementation for schools implementing under 
5 years, but not for 5 or more, a novel finding. 

Regarding differences across stage of implementation, district-level predictors were 
strongly influential for schools early in implementation but less so for schools that had been 
implementing for more than 5 years. For these schools, the strongest predictor was Team Use of 
Data, which may have allowed a level of continuous regeneration (McIntosh, Filter, Bennett, 
Ryan, & Sugai, 2010), wherein school teams adapt their implementation to fit more strongly with 
changed contexts and student populations. A unique finding of the current study is that the 
district context (e.g., the proportion of other schools) appeared to have a stronger influence early 
in the implementation process (i.e., starting to implement with a community of peers), but for 
schools that had been implementing for longer time periods, their within-school practice 
activities were stronger predictors. 

Limitations and Future Research 

There were several limitations to the current study that should be considered when 
interpreting the results. First, because the purpose of the study was to identify predictors of 
sustained SWPBIS Tier 1 systems, it is unclear whether these results can be generalized to other 
models of behavior support or even SWPBIS at Tiers 2 and 3. Second, participating schools do 


not represent a random sample. Participating districts were recruited through established state- 
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level PBIS training networks and may not reflect patterns of implementation from districts and 
schools outside of state network support. Third, years implementing SWPBIS was used as a 
proxy for implementation stage, whereas implementation science literature describes the speed of 
progress through these stages as variable and even recursive (Saldana et al., 2012). Finally, we 
measured fidelity of implementation through self-report instruments. Although these measures 
have been validated for the study purposes, future investigations might extend these results by 
including more direct measures of various factors related to both fidelity of implementation (e.g., 
direct observation of practices) and district support (e.g., staffing, dosage of coaching). 

One interesting outcome that warrants further research is that grade level became less 
predictive in later stages of implementation. This finding confirms previous research that 
SWPBIS takes more time to fully implement in secondary school settings (Nese, Nese, 
McIntosh, Mercer, & Kittelman, 2017). More research is needed to understand how SWPBIS is 
implemented and sustained in high schools. 

Implications for Practice 

Outcomes showing district-level variables to be the strongest predictors of sustained 
implementation of Tier 1 SWPBIS, along with the data showing non-malleable school-level 
factors to be much less important, indicate that focusing on establishing district capacity may be 
more promising than taking a school-by-school approach, particularly during installation and 
initial implementation. These findings also have important implications for SWPBIS trainers and 
coaches working with newly-implementing schools. Our results indicate that districts can best 
support initial and sustained implementation of behavior support practices by providing training 
and ongoing coaching in critical features of Tier 1 practices to increase the likelihood of 


implementing with fidelity as early after initial training as possible (and thus provide visibly 
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improved student outcomes that reinforce implementation). As schools achieve adequate fidelity, 
it appears key to provide training and coaching on establishing efficient systems to collect, 
review, and use fidelity and student discipline data for continuous improvement. To ensure 
maximal use of data for decision-making, district teams can focus training and coaching on 
supporting in school-level team functioning, such as conducting team meetings with consistent 
agendas, efficient decision-making processes, and an ongoing action plan (Newton, Horner, 
Algozzine, Todd, & Algozzine, 2012). Such efforts appear to be the most promising avenues for 


sustaining effective practices in schools. 
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Table 1 


School Demographic Data 


; ; ‘ 
Demographic Variable (N =860) Mean or % (SD) 
Enrollment 847 540.14 (336.92) 
% of Students Receiving FRL 847 52.0% (24.4%) 
% of Non-white Students 847 41.6% (30.8%) 
Grade Level 844 

Primary 559 66.2% 

Middle 168 19.9% 

High 96 11.4% 

Other 21 2.5% 
Urbanicity 847 

Rural 186 22.0% 

Town 119 14.0% 

Suburb 280 33.1% 

City 262 30.9% 
Years Implementing SWPBIS 860 

0 to 1 Years 212 24.7% 

2 to 4 Years 411 47.8% 

5 or more 237 27.6% 


Note. School demographic data obtained from National Center for Education Statistics for 98% 


of schools. * FRL = free and/or reduced lunches. 
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Table 2 


Means and Proportions of Missing Data by Implementation Stage 


Initial Institutionalization Ongoing Evolution 
Implementation (n= 411) (n = 237) 
(n = 212) 
Prop. Prop. Prop. 

Variable M Missing M Missing M Missing 
Criterion 

Fidelity: Y3 0.60 .009 0.63 .024 0.64 .034 
School Characteristics 

Elementary School 0.59 .024 0.69 O17 0.67 O17 

Urbanicity 2.47 014 2.86 015 2.74 O17 

Enrollment 516.50 014 $35.10 015 568.20 O17 

Prop. Non-White students 0.37 014 0.45 015 0.42 O17 

Prop. Eligible for Free or 0.53 014 0.53 015 0.51 O17 

Reduced Lunch 

Practice Characteristics 

Fidelity: Y1 0.46 .005 0.65 019 0.78 O17 

School Priority -- 132 0.02 .085 0.33 025 

Team Use of Data -- 148 0.13 042 0.47 041 

District Priority -- 107 0.02 .085 0.09 089 

Capacity Building -- .096 -0.06 .044 0.02 021 
District Characteristics 

Critical Mass 0.36 014 0.48 015 0.56 O17 

Initiative Health 0.16 014 0.04 015 0.00 O17 


Note. Y1 = Study Year 1, Y3 = Study Year 3, Prop. = Proportion. -- = Latent means were 
constrained to zero because Initial Implementation was the reference group. Proportion of 
missing data for latent variables was based on the average proportion of missing data for the 


items in the factor. 
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Table 3 


Results of Multi-Group Predictive Structural Equation Model 


Initial Institutionalization Ongoing Evolution 
Implement (n= 411) (n = 237) 
ation 
(n= 212) 
Coefficient Dp Coefficient Dp Coefficient Dp 

School-Level Predictors! 

Elementary School .10 ATT 17 .001** 04 734 

Urbanicity 04 i233 -.01 .910 -.08 410 

Enrollment -.05 623 -.05 433 -.17 12) 

Prop. Non-White -.09 564 .03 761 Al 112 

students 
Prop. Eligible for Free or -.08 479 -.09 .233 -.17 .196 
Reduced Lunch 

Practice-Level Predictors! 

Fidelity: Y1 28 <,001*** 29 = <.001*** wo .004** 

Team Use of Data 23 .004** 14 .018* 27 =<.001*** 
District-Level Predictors! 

Critical Mass 34 .006** 38 =©6<.001*** 17 158 

Initiative Health 58 =<001*** 34 = =6<.001*** .24 .027* 
School-Level Covariates? 

School Priority -.05 426 -.01 831 -.03 .643 

District Priority -.15 075 -.05 442 .03 778 

Capacity Building -.05 701 -.02 881 13 195 
R?: Fidelity Y3 45 <001*** 46 <001*** 42 <,001*** 


Note. ‘Reported coefficients are standardized beta weights. "Reported coefficients are partial 
correlations with Year 3 Fidelity. Yl = Study Year 1, Y3 = Study Year 3. * p< .05, ** p<.01, 


**®  < O01. 
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Figure I. Model predicting sustained fidelity of implementation in Year 3 of the study. 
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Note. Solid line = path statistically significant for all 3 implementation groups. Dashed line = 
path statistically significant for Initial Implementation and Institutionalization groups. Dotted 
line = path statistically significant for Institutionalization group only. Non-significant paths and 


residuals removed for clarity. 


